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CROSS-REFERENCE TO RELATED APPLICATIONS 

This application is related to a family of patent applications filed on the same day by the 
same inventors including those entitled: (1) Automated Statistical Self-Calibrating Detection 
and Removal of Blemishes in Digital hnages Based On Multiple Occurrences Of Dust In 
Images; (2) Automated Statistical Self-Calibrating Detection and Removal of Blemishes in 
Digital Images Based on a Dust Map Developed From Actual Image Data; (3) Automated 
Statistical Self-Calibrating Detection and Removal of Blemishes in Digital Images Dependent 
Upon Changes in Extracted Parameter Values; (4) Automated Statistical Self-Calibrating 
Detection and Removal of Blemishes in Digital Images Based on Determining Probabilities 
Based On hnage Analysis Of Single Images; (5) Determination Of Need To Service A Camera 
Based On Detection of Blemishes in Digital Images; (6) Method Of Detecting and Correcting 
Dust in Digital Images Based On Aura And Shadow Region Analysis; (7) Digital Camera; and 
(8) Digital Image Acquisition And Processing System, which are all hereby incorporated by 
reference. 

BACKGROUND 

1. Field of the Invention 

This invention related to digital photography and in particular, automated means of 
removing blemish artifacts fi-om images captured and digitized on a digital process. 

2. Description of the Related Art 

Many problems are caused by dust in particular and blemishes in general on imaging 
devices in general and digital imaging devices in particular. In the past, two distinct forms of 
image processing included providing means to detect and locate dust, scratches or similar defects 
and providing means to remedy the distortions caused by the defects to an image. It is desired to 



GrayCary\SF\3093797.1 
2100874-991350 



-3- 



Express Mail: EV 330728515 US 



Atty. Docket: 2100874-991350 



have an advantageous system that combines these functions, and can automatically detect and 
correct for the effects of dust, scratches and other optical blemishes. 

Dust has been a problem in scanning devices for a long time. Various aspects of the 
scanning process, enhanced by image processing techniques where appropriate, have been 
employed to provide means for the detection of dust or defects relating to document or 
image/fihn scanners. These devices form an image by moving a ID sensor pixel array across a 
document platen, or in some cases, the document platten, with document/image is moved in 
under the sensor array. The physics and implementation of these devices differ significantly 
from those of a field-based sensor or camera-style device. It is desired particularly to have dust 
and/or blemish detection and correction techniques for field-based or camera-style acquisition 
devices. 

Image correction has been studied in relation to display devices, output apparatuses such 
as printers, and digital sensors. Image correction of dust artifacts can be used to recreate missing 
data, also referred to as in-painting or restoration, or undoing degradation of data, which still 
remains in the image, also referred to as image enhancement. It is desired to have a system 
including a digital camera and an external device or apparatus that can facilitate a defect 
detection and/or correction process involving sophisticated and automated computerized 
programming techniques. 

SUMMARY OF THE INVENTION 

A method of automatically correcting dust artifact regions within images acquired by a 
digital image acquisition system including a digital camera with a lens assembly is provided. 
The method includes determining probabilities that certain pixels within multiple digitally- 
acquired images correspond to one or more dust artifact regions. A statistical dust map is formed 
including mapped dust artifact regions based on the dust artifact determining. The mapped dust 
artifact regions are separated into aura regions partially obscured by dust and shadow regions 
substantially obscured by dust inside the aura regions. Digital data are corrected corresponding 
to pixels within one or more digitally-acquired images including dust artifact regions separated 
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into aura regions and shadow regions correlated with mapped aura and shadow regions of the 
statistical dust map. 

The dust artifact region probability determining including separating suspected dust 
artifact regions into aura regions and shadow regions, and determining separate probabilities of 
whether the aura regions and the shadow regions of the suspected dust artifact regions in fact 
correspond to dust artifact regions within the images. The determming with respect to a shadow 
region may be based on an extracted parameter-dependent shadow region analysis. The shadow 
region analysis may presume that certain regions on a sensor of the digital image acquisition 
device are fully obscured by the dust. The shadow region analysis may include calculating 
effects of differences in values of the one or more extracted parameters in different images on 
dust artifact illumination, shape, position, reflection or transmission properties, distance of dust 
to the sensor, aperture, exit pupil, or focal length, or combinations thereof. The different images 
may have been acquired of different objects. The determining with respect to an aura region 
may be based on an extracted parameter-dependent aura region analysis. The aura region 
analysis may presumes that certain regions on a of the digital image acquisition device are 
partially obscured by said dust. The aura region analysis may include calculating effects of 
differences in values of the one or more extracted parameters in different images on dust artifact 
illumination, shape, position, reflection or transmission properties, distance of dust to the sensor, 
aperture, exit pupil, or focal length, or combinations thereof. The different images may have 
been acquired of different objects. 

The correcting operation may include in-painting or restoration, or both. The correcting 
may include in-painting the shadow region. The in-painting may include determining and 
applying shadow region correction spectral information based on spectral information obtained 
from pixels outside the shadow region. The correcting may include restoration of the aura 
region. The restoration may include determining and applying aura region correction spectral 
information based on spectral information obtained from pixels within the aura region. 

The correcting operation may include calculating the aura region and the shadow region. 
The correcting operation may further include correcting a shadow region approximately based on 
the correcting of the aura region. The dust artifact region probabiUty determining may be based 
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at least in part on a comparison of suspected dust artifact regions within two or more images. 
The comparison may include a first comparison of the shadow regions and a second comparison 
of the aura regions. The determining may include statistically combining probabilities based on 
the comparisons of the shadow regions and of the aura regions. 

The dust artifact region probability determining may be based at least in part on a 
comparison of suspected dust artifact regions within images with predetermined characteristics 
indicative of the presence of a dust artifact region. The comparison may include a furst 
comparison of shadow regions and a second comparison of aura regions. The determining may 
further include statistically combining probabiUties based on the comparisons of the shadow 
regions and of the aura regions. 



BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates a mam workflow of a dust removal process in accordance with a 
preferred embodiment. 

Figure 2a illustrates the creation of a dust map. 

Figure 2b illustrates an altemative embodiment of the creation of a binary dust map. 
Figure 3 outlines a correlation of a dust map to image shooting parameters. 
Figure 4a illustrates a procedure for detecting and removing dust &om images in accordance 
with a preferred embodiment. 

Figure 4b illustrates reconstruction in-painting of dust regions. 

Figure 4c illustrates the occurrence of dust in a picture with high edge information. 

Figure 4d illustrates the numerical gradient in an image. 

Figures 4e-4f illustrate a spiral procedure of in-painting from the periphery inwards. 
Figure 5 illustrates a quality control metiiod of checking the dust map. 
Figures 6a-6f represents an optical geometry of a lens that may be used in accordance 
with a preferred embodiment: 
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Figures 6a, 6b and 6c represent three lenses with the same, fixed, focal distance, with the 
same focal number, but with different constructions. 

Figure 6d illustrates the concept of an exit pupil and the distance to it. 

Figure 6e illustrates the intersections of the principal ray with windows and image plane, 
which is preferably approximately the sensor plane. 

Figure 6f illustrates the shift of one dust spec in comparison oa a shift of another dust 
spec in an image. 

Figures 7a-7g generally illustrate effects of dust on the creation of an image using a 
known optical system: 

Figure 7a illustrates an influence of a dust particle located on the input surface of the 
window on the beam converging towards the imaging point. 

Figure 7b illustrates a side view of the rays as obscured by dust as a fimction of the 
aperture. 

Figure 7c illustrates a frontal projection of the rays as obscured by a dust as a fimction of 
the aperture. 

Figure 7d and Figure 7e illustrate a power distribution map of the same dust spec as 
manifested in different f-stops. 

Figure 7f illustrate an effect of the change in focal length on the area obscured by the 

dust. 

Figure 7g illustrates the direction of movement of dust as a function of the change in 
focal length in relation to the center of the optical axis. 

Figure 8 illustrates an adjustment of the dust map based on aperture. 
Figure 9 illustrates an adjustment of a dust map based on focal length. 

Figure 10 illustrates a process of estimating based on empirical data the values of the 
parameters in the formulae that defines the change in dust as a fimction of change in focal length. 

Figure 1 1 illustrates a process of estimating based on empirical data the values of the 
parameters in the formulae that defines the change in dust as a fimction of change in aperture. 

Figure 12 illustrates a fiirther process of in accordance with another embodiment. 
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BRIEF DESCRIPTION OF TABLES 

Table 1 lists parameters in mathematical formulations of the optical system. 
Table 2 Usts potential Extracted Lens Parameters. 

DETAILED DESCRffTION OF THE PREFERRED EMBODIMENTS 

SOME DEFINITIONS 

Dust specs: The preferred embodiment takes advantage of the fact that many images may 
have repetitive manifestation of the same defects such as dust, dead-pixels, burnt pixels, 
scratches etc. Collectively, for this specifications all possible defects of this nature are referred to 
in this application as dust-specs or dust defects. The effects of those dust specs on digital images 
are referred to herein as dust artifacts. 

Acquisition device: the acquisition device may be a a multi-fimctional electronic 
appUance wherein one of the major functional capabilities of said appliance is that of a digital 
camera. Examples can be include digital camera, a hand held computer with an imaging sensor, 
a scanner, a hand-set phone, or another digital device with built in optics capable of acquiring 
images. Acquisition devices can also include fikn scanners with a area-capture CCDs, as 
contrasted, e.g., with a Une scan mechanism. D-SLR: Digital Single Lens Reflex Camera. A 
digital camera where the viewfmder is receiving the image firom the same optical system as the 
sensor does. Many D-SLR, as for SLR cameras have the c^ability to interchange its lenses, this 
exposing the inner regions of the camera to dust. 

A few parameters are defmed as part of the process: 
N Number of images in a collection. 

HIdp Number of images for occurrence to be high dust probability 

HSdp Number of recurring specs for label a region to be high dust probability 
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p(hsdp) Probability threshold for high confidence of a dust spec . 

N dp Number of images to determine that a region is not dust 

p(ndp) Probability threshold to determine that a region is not a dust region. 

Most likely Hidp <= HSdp 
I - is a generic image 

I(x,y) pixel in location x horizontal, y vertical of Image I 

pM is a continuous tone, or a statistical representation of a dust map 

dM is a binary dust map created form some thresholding of pM. 

MATHEMATICAL MODELING OF THE OPTICAL SYSTEM 

Prior to understanding the preferred embodiments described herein, it is helpful to 
understand the mathematical modeling of the camera optical system. With this modeling, the 
preferred embodiments may advantageously utilize a single dust map for dust detection and/or 
correction techniques rather than creating a dust map for each instance of dust artifact having its 
own shape and opacity depending on extracted parameters relating to the imaging acquisition 
process. With an ability to model the optical system and its variability, a single map may suffice 
for each lens or for multiple lenses, and for multiple focal lengths, multiple apertures, and/or 
other extracted parameters as described in more detail below. 

In order to study the shadow of an object on a plane, it is helpful to consider the 
following: 

- the illumination of the object (the spectral and coherent characteristics of the light) 

- the shape of the obj ect (including its micro-geometry) 

- the reflection and transmission properties of the object 

- the relative position of the object and the plane 

The case of interest for defining the model of dust as it appears on the sensor, is that of an 
object close to the image plane of a lens, which is a complex optical system, with natural light 
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illumination. The object shape and its reflection and transmission properties are practical 
impossible to model because no any specific information on the dust particle is available. On the 
other side, because the dust particles are small, it is very probable to have the same reflection and 
transmission properties on their surfaces. The distance between the dust and the sensor, which is 
the image plane is small and is in the order of magnitude of fraction of a millimeter. 
Some Definitions as to notations are now provided: 



Pe- 


exit pupil position, the distance between the sensor, and the Exit Pupil 


tw - 


thickness of the window, distance of dust to the image plane 


f- 


focal length of the objective 


m- 


focal number 


(u,v) - 


denote the coordinate system of the input surface, having its origin in the 
intersection point with the lens optical axis. 


(x,y)- 


the dust position on the image plane in the coordinate system in the image 
plane with the origin in the intersection point with the lens optical axis. 


l(x.y)' 


be the value of the irradiance in the image point (x,y) 


D - 


is the exit pupil diameter 


ho 


is the distance of the objects on the dust plane to the center of the optical 
path 


h 


is the distance form the object to the center of the optical axis 


hk 


is the distance of dust spec k on the dust plane to the center of the optical 
path 



Table 2. Parameters in mathematical formulation of the Optical system. 



Figure 6a, 6b and 6c represent three lenses with the same, fixed, focal distance, with the 
same focal number, but with different constructions. The first construction, in figure 6-a is the 
most common. The second in figure 6b is specific for the lenses used in metrology. This type of 
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lens is called telecentric in the image space. The third in figure 6c construction is rarely found in 
optical systems, but not impossible 

Figure 6d illustrates the concept of an exit pupil and the distance to it. The exit pupil 
distance, 644, is the parameter that defines this property, and represents the distance fi-om the 
secondary principal plane, 646, of the lens and the intersection of the axis of the light cone and 
optical axis of the lens. The focal distance 644 represents the distance form the secondary 
principal plane to the object plane. 

In the case of the second type lens, as defined in Figure 6b, the exit pupil position is at 
infinity. In the first and third case, figures 6a and 6c respectively, the intersections of the Ught 
cone with the dust plane are ellipses. In the case of the second lens type. Figure 6b the 
intersection of the Ught cone with the dust plane is a circle. 

In the case of zoom lens, the exit pupil position can be constant (at infinity), or can vary 
significantly, dependmg on the construction of the lens. 

This information about the exit pupil is usually not public because, in general, it is not 
usefiil for the common user of the photographic or TV lenses. However, this information can be 
readily available with some measurements on an optical bench. Altematively, this information 
can be achieved based on analysis of dust in the image. 

Figure 6e illustrates the Intersections of the principal ray with windows and image plane, 
which is the sensor plane. This figure illustrates the various parameters as defined in Table 1. 
Figure 7a describes the influence of a dust particle (obscuration) located on the input surface of 
the window (obscuration plane), on the beam converging towards the imaging point. 

Figure 7b illustrates the side view of the rays as obscured by a dust as a function of the 
aperture. Figure 7c illustrates the frontal projection of Figure 7-b, of the rays as obscured by a 
dust as a fiinction of the aperture. Figure 7d and Figure 7e illustrates a power distribution map of 
the same dust spec as manifested in different f-stops, namely a relative open aperture f-9 for 
Figure 7d and a closed aperture, namely f-22 and higher, in Figure 7e. Once can see that the dust 
spot as recorded with the closed aperture, is much sharper and crisper. 

The dust particle is situated in the window plane, and totally absorbing or 
reflecting the portion of the beam (S2) that normally reaches the image point (P). Let I(x,y) be 
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the value of the irradiance in the image point (x.y) in the absence of the dust particle. Assuming 
that the energy in the exit pupil of the optical system is uniformly distributed, then, in the 
presence of the dust particle, the I(x.y) will be reduced proportionally with the ratio between the 
area of the intersection of the light beam (SI) with ttie clean zone of the window, and the whole 
area of the same intersection (Sc). 

Remember that Sc is a function of the lens f-number (£'#) and window thickness tw 
So, the value of the real irradiance in the presence of dust will be 



or, by taking into account that SI = Sc - S2 and S2 = Area of intersection between the dust 
particle and the illuminated area: 



Because the dust particle is small, the next assumption can be made: 

For all the image points affected by a specific dust particle, the area Sc remains constant. 
In the case of the telecentric illumination, this assumption holds true for all the image points. 

With the assumption above, we can correlate the dust geometry, position and shadow. 

First of all, we now study the distribution of the irradiance on the input plane of the 
window. The intersection of the conic beam with this plane is generally an ellipse. The major 
axis of this elUpse is along the intersection of the plane mentioned above and the plane 
determined by the image point and the optical axis of the lens. 
The minor axis is 



whereas the major axis is 



r(x,y)=l(x,y) 



SI 
Sc 



(4) 




(4-a) 
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D_ 1^ 

' cos 



(6) 



where D is the exit pupil diameter , so 



D = 



£_ 



(7) 



In order to reach a complete formula of the irradiance distribution we now stress the assumption 
that for all image points affected by a chosen dust particle, "a" varies insignificantly 
Let (uo ,vo) be the point of intersection of the principal ray with the plane mentioned above. 
So, the characteristic function of the illuminated zone will be 



C = C(u,V,UQ,VQ,f\f/#,P„tJ 



(8) 



Let D(u ,v) be the characteristic function of the dust .Then, the center of the cone 
intersection with the input plane of the window will be given by (3). 
Thus, 



Sdoc = jlD{u,vyc{u,v,u,,v„f\fm,P„t^)dudv 



(9) 



R xR 



lix,y) = l(x,y) 



the relation (3), yields 
f 



llD(u,v)*C(u,v,u„v„f\f/#,P,,t^)dudv 



RxR 



Sc 



(10) 

which eventually, using 



r(x,y) = l{x,y) 



1- 



1\d(u,v)*C 

RxR 


M,V, 


I P.) 




1+^ 


y,r,f/#,P,,t„ 

J 


dudv 


Sc 



(11) 
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The terms (HtJP^x and (1+tJP^y determine the "movement" of the dust shadow with the 
change of the focal distance and, implicitly, of the Pe, as explained later on in equation 13. 
In the case of the telecentric illumination 

- Pe is infinite , 

- uO = X and vO=y. 

- the ellipse becomes a circle 

Qualitatively, as described quantitatively in equation 1 1, the fall off, or the diminished 
effect of the dust on an image varies as follow: The dust is becoming less significant as the 
aperture becomes larger, or the f-stop is smaller and the pixels are inversely affected based on 
the distance of the pixel to the periphery of the dust. 

Figure 7f illustrates the adaptation between the dust map and the image based on the 
focal length of the lens. In this figure one can qualitatively see that the area covered by the dust 
spec will be shifted as a function of the focal length. 

Also, information about the spatial distribution and/or orientation of dust particles within 
the dust map may be used to assist with determining probabilities that regions are either dust or 
non-dust. As an example, one could expect that a global average of the orientation of elongated 
dust particles would average to zero. However, at a more localized level, charged and/or 
magnetized dust particles can tend to align with similar orientations due to local dielectric forces. 
This effect would be partly dependent on the size, overall shape distribution (round or more 
elongated) and the overall distribution (uniform or clumped) of the dust regions in the dust map. 
This can tend to be particularly helpfiil when a decision is to be made as to whether to eliminate 
certain regions. E.g., most of the dust may be aUgned m a radial direction relative to the center 
of the sensor, and then particles in a concentric alignment are less likely to be dust. 
Given the definitions above, a more quantitative model can be deducted. 




(12) 
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because the ratio tJPe is small. Note that Pe has in common cases a negative value. 

Let (u,v) denote the coordinate system of the input surface, having its origin in tiie 
intersection point with the lens optical axis. Similarly, let (x,y) be the coordinate system in the 
image plane with the origin in the intersection point with the lens optical axis. 
Thus, if the principal ray of the beam intersects the input window plane in the point (m,v), the 
point in the image plane on the same principal ray will be The inverse transformation of (2) 



t 

1— ^ 0 

P. 

0 1--^ 



' c J 



(2) 



with the same assumption of a small tw/Pe ratio will be 



(13) 



1 + ^ 0 
0 1 + ^ 



Figure 7j depicts how points will shift as a fimction of the proximity to the optical center. 
Basically the farther the dust is form the center, the larger the displacement is. However, the 
direction and size of the displacements can be estimated. At the same time, the closer the dust 
spot is to the center, the more its shape will change. 

There are a few parameters that are not published. The distance, Tw, of the dust to the 
image plane is typically fixed for the camera and a single measurement can achieve that. The 
exit pupil distance may vary depending on the focal length of the lens, in particular for a zoom 
lens, the Exit pupil distance may not be most likely fixed, and can vary up to a telecentric mode 
where the distance is infinity. 
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However, such information can be empirically extracted by analysis of images taken by a 
known camera and lens, as illustrated below and in Figure 7e. In this case, knowledge of the 
effect of one dust spec can be used to calculated the shift for any other dust spec in the image. 

In this figure a hypothetic zoom lens of 70mm-210mm is illusfa-ated: 

- Pe-70 is the exit pupil position for the lens at 70mm, 

- Pe-210 is the exit pupil location when the lens is at its highest enlargement 210mm. 

- k, is known dust 

- m is hypothetical dust 

- hk is h for a specific dust particle k. 

The knowledge of the shift of dust image (preferable near to one of the comers of tiie 
image). The equation (12) 

/» = /«o[^l-^j (12) 

be written for a specific dust particle (k)and underline tiie dependence witii tiie focal 



can 
distance. 



Ok 



Peif)) 



(12-a) 



or 



\ 

independent fromk (12-b) 



So , if we know the evolution of the image of the "k" dust particle we can find the evolution of 
the "m" dust image 

The problem with this formula is again we don't know the positions of the dust particles 
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Writing the upper formula for a given focal distance , say/o ,we have 

(12-d) 

f^ok Wo) 
So , finally 



Kk Kifo) 



This relation do not need the exit pupil dependence , the thickness of the window and the 
dust position. Figures 7-g, illustrate the phenomena visually. Based on the above formulae, and 
in particular 12-e and 13, one can see that the dust movement is not constant, as a function of the 
focal length change, but is dependent on the distance of the dust to the center of the optical path. 
Qualitatively, the farther the dust is form the center, the larger the displacement will be. In other 
words, dust closer to the periphery of the image will have larger movement than the dust in the 
center. 

Referring to figure 7-g which describes the movement of two hypothetical dust spots, 
790 and 794 having a different distance, 792 and 796 respectively, to the center of the optical 
path 780. The movement of each pixel in always on the line between the pixel and the center of 
the optical path. This is based on the linear relationship depicted in equation 13. Thus the 
movement of the dust edge of dust spec 794 is along the lines depicted in block 799. Based on 

assumed coefBcients for equation 13, dust spec moves be a fixed ratio 1 + y , which can be 

e 

positive or negative, which means move inwards towards the center or out of the center. In this 
figure, dust spec 794 will move to become 795 while dust spec 791 will move to become 791. 
because the movement is linear, because the distance of dust 794 to the center depicted by 796 is 
larger than the corresponding distance depicted by block 792, the movement of the dust 794 will 
be larger. 

Alternatively, given formula (13) and given an image where the dust can be detected, 
one can calculate Pe as follows: 
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— — — = 1 + 

M ~ V P, 



(13-a) 



or 



/l=(l + Ux- = (l + Ox- 

X y 



(13-b) 



In summary of the mathematical model of the optical system for dust, in the general 
case, the way the image is affected by the dust depends on: 

- exit pupil position Pe 
-thickness of the window tw 

- focal length of the objective f 

- focal number 

- the dust position on the image plane (x,y) 

This can be calculated by knowing the optical system's components: 

a) the window thickness (tw), 

b) thefimctionPe(f) 

c) and 

d) the coordinates (u,v) of the dust if we want to .determine the dust image 
position on the image plane 

The main workflow of detecting and removing the dust from an image is illustrated in 
Figure 1. The preferred embodiment is activated in four different cases. In general, this preferred 
embodiment works for removing dust form a collection of images having the same acquisition 
device. Specifically, a user may acquire a picture on her digital camera (as illustrated in Block 
101). Altematively (102), a user may open a single image on a external device such as a personal 
computer, open (103) a folder of images on an external device or open a collection of images on 
a digital printing device (104). 
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The preferred embodiment then extracts the shooting parameters (120) . Such parameters 
include, but not limited to data about: Camera name, Lens brand, lens type, focal length at 
acquisition, aperture range, aperture at acquisition. 

In addition, some parameter, in particular on the lens and the camera may be also stored 
in the device , which is the acquisition device such as the digital camera or the processing device 
such as the personal computer or digital printer. Such information, which may include 
parameters such as exit pupil, exit pupil distance regarding the lens, or distance of dust to sensor 
for the camera. 



A table with such data may look like: 



Field 


Example of data 


Category 


Lens 

Manufacturer 


Nikon 


lens 


Lens Type 


AF 24mm-45mm f2.8-f3.5 


lens 


Focal Length 


38mm 


Acquisition data 


Aperture 


f-16 


Acquisition data 


Dust distance 


0.156mm 


Camera data 


Exit pupil 


19mm 


Lens data 


Exit pupil 
distance 


230mm 


Lens data 



Table 2 Extracted Lens Parameters 



The dust map may also include meta-data that are different the Ust of extracted 
parameters. Moreover, that which is described as being extracted parameter dependent or 
encoded with extracted parameter value data or based on a value of an extracted parameter can 
be broadened to include other meta-data than just the extracted parameters Usted in Table 2. For 
example, certain meta-data are dependent on parameters existing at the time of acquisition of the 
image, and can be camera-specific or not. The amount of ambient Ught available will depend on 
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whether there is artificial lighting nearby or whether it is a cloudy day. Discussion of meta-data 
as it relates to image acquisition is found in more detail at U.S. patent application serial number 
10/608,810, filed June 26, 2003, and is hereby incorporated by reference. 

In the case that the system deals with multiple images (as defined in 102, 103, and 104), 
the algorithm describes a loop operation on all images (110). The first step is to open a dust map 
(130) . If non exists (132) the system will create a new dust map (200) as fiirther described in 
Figure 2. In the case that the system has a few dust maps (130) the software will try to correlate 
one of the maps (300) to the image. This process is fiirther illustrated in Figure 3. In particular 
this correlation refers to the adjustment of the shooting conditions to some accepted standard. 
The acquisition conditions in particular refer to the aperture and the focal length. The correlation 
process is interchangeable and can be done by adjusting the image to the map or adjusting the 
map to the image. In some cases both acquired image and dust map should be adjusted to some 
common ground. Such an example may happen when the ma is calculated based on a aperture 
that the lens does not reach or a different lens than the one used wit different optical 
configuration. Alternatively, such as in the case of a new lens, this process (300) as called by 
130, may be used to adjust the map onto a new map and fi-om that stage onwards continue with a 
single m^. 

If no dust map corresponds with the image (140), a new dust map is created (200). When 
a dust map does correspond (140) to the image, the preferred embodiment checks if the dust 
specs as defined in the dust map are of high enough confidence level to bemg dust regions (150). 
The statistical decision as to the way such confidence level is calculated for the dust map in 
general and for the individual dust specs, is fiirther discussed in Figure 2 and Figure 3. If the 
confidence level is low, the image is added to the updating of the dust map (200). If after the 
image is added, the confidence level is high enough (152) the software continues to the dust 
removal process (160). Otherwise, the software progresses to the next image (170). 

For example, a dust map is considered vaUd after 10 images are analyzed, and a dust spec 
is considered vaUd after a dust is detected in 8 images. In this case, after analyzing 9 images, the 
software may continue to the stage of updating the dust map (200) but upon completion (10 
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images) there is a valid dust map, and the software will continue to the correction (160) stage. If 
however the loop (110) is only on its 1*' to 8* raiage, no correction will be done. 

As an additional embodiment, images can be corrected retroactively after the dust map 
reached high enough confidence. This can be used for batch processing or offline processing or 
processing where the information is gathered in parallel to the needed correction, hi this 
embodiment, when the confidence level is not enough (152, NOT-YET) the images, a pointer to 
them, or a Ust of them, or a representation of them, that were used to create the dust map are 
stored in temporary location (154) and when the dust map is ready (15 1-YES) , the software will 
retroactively remove the dust firom all those images (156). Li this fashion, all images, including 
ones that originally did not hold sufficient statistical information, may be corrected. 

Referring to the dust detection and correction process (160). This process may be 
executed as a loop on every dust spec in the dust map, with a stage of detection and correction 
(400, followed by 166 and 168). Alternatively, the process can be implemented where all dust 
specs are detected first (162) and then all dust specs are corrected (164). The decision as to the 
sequence of operations varies based on implementation criteria such as what regions of the image 
are in memory, and should not affect the nature of the preferred embodiment. As part of the 
detection, the software also performs a self testing (500) where each new image introduced is 
compared to the dust map. This process is fiffther defined in Figure 5. The importance of this 
stage for each dust spec and for the dust may, is that m this manner, if the dust situation changes, 
such as a single spec moving aground or the camera being serviced), the software will 
immediately detect the change and re-vaUdate the dust map. As described above, the validity test 
(500) can be implemented on a dust spec by dust spec or on the fiiU dust map. 

Referring to Figure 2-a where the Dust map creation and updating is defined: 
This process can receive a collection of images as defined by Figure 1 . blocks 108 and 109) or 
one image at a time is refereed to this process, as defined by Figure 1. block 1 10. 
If the function is called with a single image (220-SINGLE MAGE) the image is directly 
provided to the calculations (270). If multiple images are provided (240 MULTIPLE IMAGES), 
then an initial step is to define if there are more than enough images for defining the map. This 
step is designed to optimize the creation process for dust in case of large amount of images. 
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The sequence of the images that are to be referenced based on the original collection of 
N images as defmed in Figure 1 blocks 102, 103 or 104. The sequence of images is based on a 
few criteria such as: giving more weight to images shot last, and if images are shot in a relatively 
small time frame, allocate the sequence is large distances to try and assure minimal 
repetitiveness between similar images that may have been taken of the same object with little 
movement. The sequence will not be limited to the number of images (HIDP) because it may 
well be that some regions will not have enough data in them to evaluate the dust. This may 
happen in cases where part of the image is very dark in some of the images. 

As an example: if N (number of images in a selection) = 30; and HSdp (number of 
images needed to determining map) = 10; and all images were shot in a space of an hour; then 
a potential sequence may be: 

30,27,24,21,18,15,12,9,6,3,29.26,25 ,2,28,25, 1 

Altematively if the same 30 images were taken over a period of a month it may be 
beneficial to select images sequentially (last one shot is the first to be calculated): 

30,29,28, 20,19 2,1 

And in some cases this process of sampling the series (270) may also decide not to use images 
that are taken too long from the last image, hi this case, for example if image 1-15 were taken in 
July and 16-30 were taken in November, this process may Umit the new map to 16-30 or even 
create two different maps, one for images I - 1 5 and the other for 1 6-30. 

In a different criteria, the parameters as extracted fmm the raiages will determine the 
sequence and the number of dust maps that are to be calculated. For example if a folder contains 
N=30 images, where 15 were taken with one camera and 15 with another, the sampling step 
(270) may create two map sets. 

Another criteria for creating a new set or checking for new dust is the type of lens. If a 
lens is changed, it means that the CCD-cavity was exposed to potential new dust. This may 
trigger a new set of images to be looked at. It may also be an indication that the camera was 
serviced, or that the photographer cleaned the camera. Of course, if there is a parameter that 
defines when a camera was serviced, this will trigger the creation of a new dust map. 
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Those familiar in the art may be able to determine the right combinations of creating this 
sequence based on the nature of the dust, the camera and the lens. The next loop (270-271) 
defines the marking of each region and the addition of the region to the dust map if it is not 
already there. There are three type of regions in an image, the first are images with sufficient 
information to detect whether they are of dust nature. As an example, dark regions surrounded by 
a light background. Other criteria may be regions with a relatively small color saturation. The 
second group are regions that are definitely non-dust. Such regions are for example all clear, or 
of high color saturation. Other regions are inconclusive such as a very dark segment of the 
image. In this case, it will be hard to detect the dust even if it was part of the image. 
Alternatively when looking for over exposed or "dead pixels" the criteria may be reversed, if the 
pixels appear as a white spec in the image. 

The criteria may be also a fimction of the acquisition parameter. For example an image with a 

open aperture may all be marked as in-decisive, because the dust may not appear on the image. 

Regions that are potentially dust are marked (292) and then added to the dust mask (294). The 
addition may be the creation of a new dust spec on the map or the modification of the probability 
function or the confidence level counter for the region. Regions that are most likely non-dust are 
marked (282) and then added to the dust mask (284). The addition may be the creation of a new 
dust spec on the map or the modification of the probability fimction or the confidence level 
counter for the region. The additions of the regions needs to be normalized to the shooting 
conditions as defmed by the dust map (300) if this step was not performed prior to entering this 
fimction, as optionally defined in Figure 1 . 

This loop continues over all regions of the image (271). Altematively (272), each region 
is compared (500) with the map dust to se if there is no case where the monotonicity is broken, 
i.e. a region that was of high probability to be dust is now non dust. Figure 2-b describes an 
altemate embodiment of the dust creation process. Block 1210 describes the preparation of the 
dust map which includes either opening an existing map 1216 or creating a new one 1214, where 
all pixels as a starting point are non dusst or WHITE. After the map is correlated to the shooting 
conditions, 300, a dust map I-dM is created in block 1220. 
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All pixels in the image, 1222, receive the following values 1224 based on the luminance 
value of the pixel: Case 

If luminance is less than DARK_^THRESHOLD, then I-pM (X,Y) = MAYBE_DUST; 
If luminance is greater than WHITE_THRESHLOD, then I-pM(x,y) = WHITE PIXEL; 
OTHERWISE I'pM(x,y) = DON'T^KNOW. 

Once all pixels are analyzed, 1220, they are then clustered, 1296 into dust regions or dust 
specs in a I-pM dust map. The next step is to create a dust map dM which is continuous based 
on the value of the individual pixels . Li the final stage 1250, the dust map is threshold by the 
predetermined value THRESHOLD to create a binary mast. 

Figure 3 illustrates the process of correlating the image to a default settings of the dust 
map. This process defines correlating the image o the dust map, the dust map to a new dust map 
of the dust map to the mage. In particular this correlation refers to the adjustment of the 
shooting conditions to some accepted standard. The acquisition conditions in particular refer to 
the aperture and the focal length. The correlation process is interchangeable and can be done by 
adjusting the image to the map or adjusting the map to the image. In some cases both acquired 
image and dust map may be adjusted to some common ground. Such an example may happen 
when the ma is calculated based on a aperture that the lens does not reach or a different lens than 
the one used wit different optical configuration. Alternatively, in case of a new lens, this process 
(300) may be used to adjust the map onto a new map and firom that stage onwards continue with 
a single map. 

To begin with, the dust map is being loaded (1 12) and the default data on which the map 
was generated is extracted (310). Such data may include the lens type, the aperture and the focal 
length associated wit the default state. In concurrence, the information form the acquired image 
(304) is extracted (320) and compared to the one of the dust map. 

A explained in the mathematical model of the optical system, the two main adjustments 
between the dust map and the image are based on focal length, and on aperture, each creating a 
different artifact that should be addressed. Knowledge of the phenomena may assist in creating a . 
better detection and correction of the dust artifact. Altematively, in a separate embodiment. 
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analysis of the image and the modification of the dust as changed by aperture and focal length, 
may provide the necessary mathematical model that describes transformation that defines the 
changes to the dust as a fimction of change in the lens type, the focal length and the aperture. 

The mathematical distortion of the dust as a fimction of the aperture is illustrated in 
figure 7a-7f The geometrical optics illustrations of the above phenomena are depicted in 
Figures 6a-6e. Referring to figure 3, after extracting the data, the following step is modification 
of the map and or the image based no focal length (900), and based on aperture (800). The 
following steps are fiuther defined in Figure 9 and Figure 8 respectively. 

Following the too step (800 and 900) the hnage and the Dust Map are considered to be 
correlated. The correlated map cM is no longer binary because it defines both the shift and the 
fall off which is continuous. Figure 400 defines the process of detecting and removing the dust 
fiom the image. The input is tiie image I is loaded, if it is not aheady in memory (404) and tiie 
correlated dust map is cM is loaded (402) if aheady not in memory. 

The process of detecting and removing tiie dust is done per dust spec. This process is 
highly parallelized and can be performed as a single path over tiie image, or in ships. The 
flexibility of performing tiie operation on the entire image, or in portions, combined with the 
correlation or as a separate process, enables a flexible implementation of tiie algoritiim based on 
extemal restiictions defined by the hardware, tiie run time environment, memory restiictions and 
processing speed. 

As defined and justified by the physical phenomena, tiie metiiod of correcting dust 
artifacts is defined based on two different operations. The former is the retouching or flie in- 
painting here regions with no data (420) , or data that is close to noise to be recreated as defined 
in 430 and later on in Figure 7e, li. The second portion of tiie correction is based on the tapering 
degradation of surrounding pixels as a fimction of tiie apertiure, as illustirated in Figure 7b,7c,7d. 
Referring to the image enhancement portions, where tiie data exists but most likely not in its fiill 
form, due to the fact that some of tiie dust affects the quality of the image, but some regions still 
contain some data (420), an image enhancement algorithm is performed on the pixels (430). In a 
simplified embodiment (432) , assuming tiiat tiie optical model is simpUfied to a degradation in 
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the overall brightness as defined by the OPACITY, the enhanced pixels will receive a value 
inversely related to the OPACITY i.e. 9432) 

OPACITY 

To maintain a check of the validity of the model, the pixels before being operated on may 
be send for validation (500) as described in Figure 5. The second portion of the image correction 
is the restoration or the inpainting (460). In this case, the area behind the dust has no relevant 
data to enhance, or if there is, this data is relatively close to the overall noise level and thus can 
not be enhanced. Therefore, there is a desire to in-paint every pixel based on the analysis of the 
surrounding region to the dust (470). In the case where the enhancement regions as defined in 
block 430 are of good quality, those pixels as well may contribute to the in-paining. Otherwise, 
they are excluded form the calculations. The decision may rely on the distance of the pixels to 
the OPAQUE ones, of the minimum level of OPACITY that was used to restore the pixels in 
block 430. The larger the OPACITY, the smaller such pixels may be relied on in the inpainting 
process. 

Figure 4b illustrates the a certain embodiment of the in-painting process. In general, each 
pixel in the obscured region, 480, is to be filled, 482, based on its surrounding pixels, hi this 
specific embodiment, the pixels are filled in based on information of the surrounding non 
affected pixels. This specific algorithm takes into account that the pixels closet to the periphery 
has a better chance to be anticipated by the external pixels. Therefore the process is done form 
the outside inwards. 

This process of spiraling inwards is also depicted in Figure 4d. In it, given a dust spec 
1400 in a grid 1410 the dust is taken a digital form, of the surrounding bounding area 1420. 
External pixels such as 1430 are not part of the dust region, while internal ones such as 1450 are. 
An algorithm tiiat works on the periphery moving inwards as defined in Figure 4b blocks 470 
is depicted in Figure 4d as follows: hi the first round all peripheral pixels, numbered form 1 to 20 
are being operated on as defined in Figure 4b, block 474. After tiiat, all of the above mentioned 
twenty pixels, as defined in Figure 4D Block 1460 are removed, according to block 476 of 
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Figure 4b, form the region leaving a smaller dust spec of shape 1470. This modified dust spec, 
has a different set of peripheral pixels numbered 21-31. After removing those pixels by the in 
painting process, block 467, a smaller dust kemel as depicted in 1480 is left with only three 
pixels 34, 35 and 36. 

The process of filling in the pixels need not be spiral, hi a different alternative, the 
process follows the pattern of the region surrounding the dust. For example lines and edges, or 
other high frequency information are being filled in prior to filling in the smooth regions. This 
approach will prevent unnecessary blurring due to the in-painting process. 

A preferred embodiment the criteria for the chosen value can be based on marinating the 
overall gradient around the pixel, 462. The justification is that in the case of a steep edge, it is not 
advisable to copy information for the surrounding pixels with the edge value. By doing so, the in 
painting process will maintain any high frequency information in the picture such as lines and 
edges. 

An illustration of that is given in figures 4c, 4d and 4e. Referring to figure 4c, the same 
dust 1400 as in figure 4b with the bounding box 1420 on a grid 1410 is obscuring a picture 
including high frequency data such as the letter A, 1440. By taking a small section of based on 9 
pixels as depicted in figure 4d pixel 1490 is surrounding by 8 pixels 1491, 1492, 1493, 1494, 
1495, 1496, 1497, 1498. 

In a simpUfication, each pixel receives the average value between the black and white 
regions. This is depicted in figure 4e. hi this case, the pixels 1490 and its surrounding 8 pixels 
1491, 1492, 1493, 1494, 1495, 1496, 1497, 1498. have a digital value of 21, 220, 250, 245, 253, 
180, 145 and 35 respectively. It is clear that the best value of pixel 1490 will come form its 
surrounding pixels with the smallest gradient. Because in practice the value of pixel 1490 can not 
be determined, but is the center of the in painting, the value of the gradient will be based on the 
value of the gradients of the pixels around it and extrapolated to this pixel. 

hi this specific example, the differential horizontal gradient between pixels 1497 and 
1493, is the largest while the gradient vertical gradient of 1495 and 1491 will most likely be the 
same. Therefore, a preferred value of 210 will be based on the extrapolated average of the two 
gradients of its top and bottom pixels. 
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Figure 500 describes a tool that may be used in various stages of the process, namely 
quality control. This process can be called from the main workflow (Figure 1) the creation of the 
dust map (figure 3) or the image restoration and enhancement (Figure 4 and 4-a respectively). 
The purpose of this took is to perform the necessary quality check to assure that pixels will not 
be wrongly classified, nor mistakenly corrected when they should not. 

The quality control test can include: For a pixel in an image which belongs to a potential 
dust (510). In case the image was not yet correlated with the dust (300) this process should then 
be done. The next step is to check whether the pixel belongs to a dust region or not. For example, 
in a small aperture the region behind the dust should be close to if not totally obscured. If not 
(530 NO) , there is a remaining desire to revive the dust map (550) or the image does not 
correspond to the dust map. This can happen when the image is form a different time, or from a 
different acquisition device. In this case, the software will create a new dust map (200) to 
correspond to the new dust or lack thereof , or inform the user that it can not correct this image. 
In case where there is not enough information (530 MAYBE), there is no conclusive evidence to 
reject the correlation and thus the process continues (580) with no action. In the case that the 
image displays information that may concur with the dust map, the software may continue (580) 
or prior to that, enhance the likehhood in the dust map (200) that the region is indeed dust. 
Figure 8 describes the adjustment of the dust map to the image acquisition parameters based on 
the aperture. Simplifying, the closed the aperture is, the crisper and more noticeable the dust is. 
In other words, for example images taken with an f-stop of f-32, the dust will be very prominent 
and opaque, while the same image taken at f-2.8 may display no visual degradation of the image. 
The correction of the image should take that information into account, to prevent over correction 
of the dust artifact. 

The acquisition information and the corresponding dust map default setup are extracted in 
blocks 326 and 312 respectively. Then, for each dust spec in the image 810, the size of the region 
that is still obscured by the dust is calculated, as defined by mathematical model. In some cases, 
when the aperture is very open, this region may decline to 0. hi others, where the aperture is still 
very close, the size may be close to the size of the dust. Alternatively, This step, 820, may be 
done as a preparation stage, and kept in a database, which can be loaded. 
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The process then splits in two. The fiilly obscured region will be marked in 834 pixel by 
pixel 832 in a loop 834, 835 and will be treated by the in-painting process as defined in Figure 4- 
a. A semi opaque dust map, is created in the loop 840, 841 for each pixel. Each of the pixels 842, 
is assigned an PCAPITY value 844, based on the mathematical model as described previously in 
Figure 7a-4d. The dust spec that is only partially attenuated will go through a inverse filtering 
of the information aheady there, as described in Figure 4 block 430, with a specific embodiment 
in block 432. The process of the inverse filtering may take into account the signal to noise ratio 
to avoid enhancing data which is not part of the original image. For example, the region around 
the dust may have a over-shoot similar to a high pass filter, which may manifest itself in tlie form 
of an aura around the dust. This aura should to be taken into account before enhancing the 
regions. 

Figure 9 describes the adjustment of the Dust Map based on the Focal length, and the 
specific lens. The scientific background is explained in figures 7e-7f. As described before, the 
shifl; of the dust spec as a fimction of focal length for a specific lens is a fimction of equation (13 
the thickness of the window- tw, which is constant for a given camera and exit pupil position- 
Pe which varies based on the lens system and the variable focal length in case of a zoom lens. 
Given an image and a dust map, the pertinent information is loaded, as described in Figure 3, 
namely the focal lens and lens type of the camera, 326, the focal length and lens type in the dust 
map 312 and the camera distance of dust to the sensor 318. 

The process then goes through all knows dust specs in the image 910 and calculates the 
shift of the dust as explained in figure 7a-7e. The coordinates of the pixel are calculated fi-om the 
center of the optical path , 922, and the shift is calculated 924. Alternatively to going through 
each pixel, in order to speed the process, only the periphery of the dust spec can be calculated 
and the rest will be filled in. Moreover, because the shift is a fimction of the location (x,y) of the 
dust, in the case where dust is far enough firom the origin, the dust shape will not change. It is 
then sufiBcient to calculate only a shift of a single point and displace the entire dust spec 
accordingly. The regions, which are only partially attenuated due to the change in aperture, may 
be calculated in this stage 940, 941, if calculated aheady as illustrated in figure 8, or 
alternatively, the displacement can be calculated first as explained in blocks 942,944. 
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In some cases, it is impossible to get the data on the exit pupil, nor the distance the dust is 
from the sensor. Such cases may be when the application has no a-priori knowledge of the 
camera or the lens that was used. 

It is still possible in those cases to create a reasonable dust map, by empirically 
reconstruction the parameter based on analysis of the images, the shift in the dust and the falloff 
of the dust edges. Such techniques are defined in Figure 10 and 1 1 defining the process of 
estimating, based on empirical data, the values of the parameters in the formulae that defines 
the change in dust as a fiinction of change in aperture . 

This process can be useful in the case that the software has no a-priori knowledge of the 
extracted parameters of the camera or lens that are used to convert the acquired image and the 
dust map image into the same known space. In an alternative embodiment, this technique can 
also be used to extract optical parameters, similar to methods done in a controlled laboratory 
settings such as using an optical bench of a lens regardless of the desire to remove dust artifact 
from an image. Alternatively this process can be used with partial knowledge, such as the shift 
of a single dust to evaluate the transposition of all dust specs in the image. 

Figure 10 Defines the process of creating the mathematical formulae based on 
empirical data to parameterize the change in dust as a fimction of change in focal length. 

In general, a proposed embodiment reUes on the fact that when a dust if found, a pattern 

t 

matching can be applied to find the shift in the dust. Based on this information, can be 

calculated as defined in Equation 13 -a. If is known then Pe can be calculated as recited in 
equation 1-b. 

Specifically, in a preferred embodiment, an image is acquired, 1010 and a dust map is 
calculated 1012. A second image is captured, 1030 with a different focal length than the first 
image, and a dust map is calculated 1032. The process repeatedly tries to find two dust spots in 
the two dust maps 1040. If no dust specs are correlated the process is repeated for consecutive 
images, 1020. 



GrayCary\SF\3093797.l 
2100874-991350 



-30- 



Express Mail: EV 330728515 US Atty. Docket: 2100874-991350 

The process of finding dust specs is calculated by applying a local correlation to each of 
the dust specs. Preferably, based on Equation 13-a, the fijrther the dust is firom the center, the 
better the precision is. 

When two specs are determined to be firom the same dust spec the disparity between the 
specs is calculated 1042. The ration between the shifted pixels is calculated. This ratio is the 
numerical empirical estimation of t„ in equation 13-a. Moreover, if the distance of the dust to 
the sensor is known , the Exit pupil of the lens can be calculated based on the same equation. 

Figure 1 1 Defines the process of estimating based on empirical data the values of the 
parameters in the formulae that defines the change in dust as a fimction of change in aperture 

The process is similar to the one described for estimation of the focal length, albeit the fact that 
the parameters calculated are different. 

Specifically, a first image is acquired, 1 1 1 0 and dust is detected, 1 120 in this image. If 
the image is not appropriate for detecting the dust, or if the probability is low for the dust 
regions, this image is rejected for this purpose and another image is chosen. For the empirical 
comparison, a second image is captured 1 140, or alternatively a set of images 1 130 , all with 
varying aperture, to enable acceptable sampling set. The process then looks for a detected dust 
region with high level of probability, hi the case that all other parameters are similar except the 
aperture, the process can search for dust regions in the same coordinates that the original image 
dust regions were found. The dust regions of the two or more images are correlated, 1 160. The 
process continues for a sufficient amount of dust regions, 1 168, which in some cases can even be 
a single one, and sufficient amount of images, 1 169, which can dso be, depending on the 
confidence level, a single image. Once the dust regions are correlated 1 160, the fall off due to the 
change of aperttire is calculated, 1 1 72, on a pixel by pixel basis, 1 1 70, for every pixel in a dust 
region, 1 179. Based on this information, the fall off function is calculated. 1 180. hi a preferred 
embodiment, the fall off fimction is determined as a fimction of the distance of a pixel firom the 
periphery of the dust, as well as the aperture. 

Alternatively, the dust specs may also be determined by trying to correlate the dust spec 
in the map to the acquired image. Such correlation can be performed within a reasonable window 
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size or even on the diagonal line between the dust and the center of the optical path, in the 
direction that the dust may move. By gaining knowledge on the movement of a single dust spec, 
as explained in formulae 13a-13d, all other dust specs shift can be determined. 

It is also possible to determine whether the camera should be physically cleaned based 
on analysis of the dust in the camera and the specific dust pattems. This illustrated in the 
flowchart of figure 12. An example will be that certain width of dust will not allow correct in- 
painting based on the surround. Another example will be the overall number of dust specs or the 
overall relative area that the dust covers. The input for such analysis is a dust map, 1200. This 
map can be similar to the dust map generated in block 200, or any other representation of a dust 
map or a calibration map, generated automatically or manually by the photographer. Such 
analysis need not be performed for every image. A process, 1210 determines whether to perform 
such analysis. Examples to trigger this process are the time since the last analysis, change of 
lenses, which may create the introduction of dust, or message form the quality control of dust 
map, as defined in Figure 5 block 500, that the dust has shifted or that the dust no longer 
corresponds to the map. In general, the any changes in the dust structure may be a justification to 
trigger the analysis process. When no analysis is desired, 1212, the process terminates. 
Otherwise, the analysis is performed, 1220. The analysis is performed for each dust spec 
mdividually, 1 130, and then the results are accumulated for the whole image. For each dust 
region, 1 140, some parameters are being extracted, including, but not limited to Area of dust 
region; Maximum width of dust region; the distance between dust region and neighboring 
regions; movement of dust region form last analysis; occurrence of new dust specs since last 
analysis; etc. Following this analysis, the results are summarized to include such data as - the 
overall area of dust, in size and in percentage for the entire image; the largest width of dust; the 
largest area of dust spec; changes in area of dust since last analysis and changes of dust particles 
since last analysis. In an alternate embodiment, either automatically or based on the 
photographers preference, this analysis may be displayed or saved in a log file for fiiture 
reference, 1221 . Each of the aforementioned criteria may have a acceptability threshold value. 
This threshold value is determined empirically by the manufacturer in terms of the maximum 
acceptable dust that can be corrected in software. Alternatively, this data may be adjusted to by 
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the photographer based on her tolerance level to dust. If any of the parameters exceeds an 
acceptable threshold, the user is informed, 1290, that the camera should be manually maintained 
and cleaned up. 

Alternatively, 1285, this same process may be used as a tool to inform the user of 
changes in the dust. Such information is particularly important in the case that the dust 
correction algorithm is based on the creation of a calibration image. In this case, the analysis 
will be used to inform the user that a new caUbration image should be acquired to support the 
correction of dust in future images. 

Alternatively, the process of analysis described above may also be incorporated in a 
maintenance procedure, where the camera, after being cleaned up will perform an auto-test, as 
described in Figure 12, to verify that the camera now is indeed clean. In such cases, the threshold 
parameters are of course substantially more restrictive and demanding, to assure high quality 
maintenance. 

There are many alternatives to the preferred embodiments described above that may be 
incorporated into a image processing method, a digital camera, and/or an image processing 
system including a digital camera and an external image processing device that may be 
advantageous. For example, an electronic circuit may be designed to detect maximum and 
minimum dust detection probability thresholds while acquiring pixels in an image (see also US 
patent 5,065,257 to Yamada, hereby incorporated by reference). Such a circuit can produce 
signals that may be used in processing to eliminate regions, which Ue outside the expected range 
of signal values due to the presence of dust particles or similar optical defects, or alternatively to 
accept, maintain or eliminate regions as dust artifact regions based on whether a probability 
determination exceeds, high or low, a certain threshold or thresholds. A technique may be used 
to detect and provide a remedy for the effects of dust on a digitally-acquired image (see also US 
patent 5,214,470 to Denber, hereby incorporated by reference). An image may be recorded 
within a digital camera or external processing device such as may be signal coupled with a 
digital camera for receiving digital output image information or a device that acquires or captures 
a digital image of a fibn image. The digital image may be compared with the fihn image, e.g.. 
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through a logical XOR operation, which may be used to remove dust spots or blemishes picked 
up in the acquisition of the digital image from the film image. 

Multiple images may be processed and stationary components, which are common 
between images, may be detected and assigned a high probability of being a defect (see also US 
patent 6,035,072 to Read, hereby incorporated by reference). Additional techniques, which may 
be employed to modify defect probability, may include median filtering, sample area detection 
and dynamic adjustment of scores. This dynamic defect detection process allows defect 
compensation, defect correction and alerting an operator of the likelihood of defects. 

Dark field imaging may be employed to determine the location of defects in digital 
images fi-om digital cameras or film scanners (see US patent 5,969,372 to Stavely et al, and US 
patent application 2001/0035491 to Ochiai et al, each hereby incorporated by reference). A 
normal imaging of a object with normal illumination may be followed by a second imaging using 
different wavelengths, e.g., infrared illumination. Dust, fingerprints, scratches and other optical 
defects are typically opaque to infrared light. Thus the second image produces an image with 
dark spots indicating the position of dust particles or other defects. 

A process may involve changing any of a variety of extracted parameters (see elsewhere 
herein), angle of sensor relative to image plane, distance of image plane or sensor from dust 
specks (e.g., on v^ndow of sensor), etc., and imaging a same object with the digital camera. A 
comparison of the images reveals with enhanced probability the locations of dust artifact. In a 
camera application, the unique location of the actual dust relative to the object and to the image 
plane provide information about extracted parameter-dependent characteristics of dust artifact in 
the images. The analysis for the digital camera application depends on the "transmission" -based 
optical parameters of the system, i.e., the fact that light travels from a scene through the camera 
lens and onto the camera sensor, and not involving any substantial reflective effects. It is 
possible to make determinations as to where the dust actually is in the system by analyzing 
multiple images taken with different extracted parameters, e.g., on the sensor window, or in an 
image of an original object which itself is being images such as in film imaging. 

In a scanning application, this technique can be use the face that a speck of dust will cast 
a shadow of a different color, geometry location, etc. with changes in extracted parameters, e.g.. 



GrayCary\SF\3093797.1 
2100874-991350 



-34- 



Express Mail: EV 330728515 US 



Atty. Docket: 2100874-991350 



with a different color with increasing elongation of the shadow for each parallel row of pixels (a 
"rainbow" shadow, as it were). Multiple scans taken from various angles of illumination may be 
employed to produce an image which identifies dust defects from their shadows and the colors 
thereof (see US patent 6,465,801 to Gann et al. and US patent applications 2002/0195577 and 
2002/0158192 to Gann et al, hereby incorporated by reference). A linear scanning element 
moves across a document (or the document is moved across the scanning element) and an image 
of the document is built up as a series of rows of pixels. This differs from the physical 
configuration of a camera in which a shutter illuminates a X-Y sensor array with a single burst of 
light. In both cases, though, dust may lie close to the imaging plane of the sensor. 

Technique may be applied as part of a photofinishing process to eliminate blemishes on a 
film image obtained by a digital camera (see also US patent appUcation 2001/0041018 to 
Sonoda, hereby incorporated by reference). Such techniques may import previous acquired 
information about defects in images from a blemish detection procedure. A technique for 
correcting image defects from a digital image acquisition device such as a digital camera may 
involve repeated imaging of an object or other image, where each successive image-acquisition 
involves different properties or extracted parameters or meta-data related properties, such as 
variable angles of incidence or variable lighting or contrast parameters, and the results of these 
repeated scans may be combined to form a reference image from which defect corrections are 
made (see also US patent application 2003/01 18249 to Edgar, hereby incorporated by reference). 

A decision on whether a defect in a image acquired by a field-based digital camera is to 
be corrected or not may be based on a balancing of considerations. For example, the likely 
damage to surrounding defect-free portions of the image may be balanced against the likelihood 
of successfiiUy achieving correction of the defect. 

Image processing means may be employed where the detection or correction of defects in 
a digital image may be based solely on analysis of the digital image, or may employ techniques 
directly related to the image acquisition process, or both. Anomalous unage regions may be 
determined based on the difference between the gradient of an image at a set of grid points and 
the local mean of the image gradient (e.g., see US patent 6,233,364 to Krainiouk et al., hereby 
incorporated by reference). Such technique can reduce the number of false positives in "noisy*' 
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regions of an image such as those representing leaves in a tree, or pebbles on a beach. After 
detennining an initial defect Ust by this means, the technique may involve culling the Ust based 
on a one or more or a series of heuristic measures based on color, size, shape and/or visibility 
measures where these are designed to indicate how much an anomalous region resembles a dust 
fragment or a scratch. 

Techniques and means to correct scratches in a digitized images may employ a binary 
mask to indicate regions requiring repair or noise removal, and sample and repair windows to 
indicate (i) the region requiring repair and/or (ii) a similar "sample" area of the image (see also 
US patent 5,974,194 to Hirani et al., hereby incorporated by reference). Data from a sample 
window may be converted to a frequency domain and combined with frequency domain data of 
the repair window. When a low-pass filter is applied, it has the effect to remove the sharp, or 
high-frequency, scratch defect. 

Techniques and means of detecting potential defect or "trash" regions within an image 
may be based on a comparison of the quadratic differential value of a pixel with a pre- 
determined threshold value (see US patent 6,125,213 to Morimoto, hereby incorporated by 
reference). The technique may involve correcting "trash" regions within an image by 
successively interpolating from the outside of the "trash" region to the inside of this region. 

Techniques and means to automate the removal of narrow elongated distortions from a 
digital image may utilize the characteristics of image regions bordering the distortion (see also 
US patent 6,266,054 toLawton etal., hereby incorporated by reference). User input may be 
used to mark the region of the defect, or automatic defect detection may be employed according 
to a preferred embodiment herein, while the process of delineating the defect is also preferably 
also performed automatically. 

Techniques and means to allow automatic alteration of defects in digitd images may be 
based upon a defect channel having a signal proportional to defects in the digital image (see also 
US patent 6,487,321 to Edgar et al., hereby incorporated by reference). This allows areas of 
strong defect to be more easily excised without causing significant damage to the area of the 
image surrounding the defect. 
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Techniques and means may be employed to generate replacement data values for an 
image region (see also US patent 6,587,592 to Georgiev et al, hereby incorporated by reference) 
Image defect may be repaired as facilitated by the replacement data. Moreover, the repairing of 
the unwanted image region may preserves image textures within the repaired (or "healed") 
region of the image. 

Techniques and means may be employed to detect defect pixels by applying a median 
filter to an image and subtracting the result firom the original image to obtain a difference image 
(see also US patent application 2003/0039402 and WIPO patent application WO-03/019473, 
both to Robins et al, each hereby incorporated by reference). This may be used to construct at 
least one defect map. Correction of suspected defect pixels maybe achieved by replacing those 
pixel values with pixel values from the filtered image and applying a smoothing operation. User 
input may or may not be utilized to fiurther mitigate the effects of uncertainty in defect 
identification. 

Techniques and means for retouching binary image data which is to be presented on a 
view-screen or display apparatus may be employed to eliminate local screen defects such as dust 
and scratch artifacts (see also US patent application 2002/0154831 to Hansen et al., hereby 
incorporated by reference). The production of visible moire effects in the retouched image data 
may be avoided by the replacement of small areas. 

A digital video camera with sensor apparatus may incorporate a defect detecting mode 
(see also US patent 5,416,516 to Kameyama et al., hereby incorporated by reference). The 
locations of detected defect pixels may be retained in the memory of the camera apparatus and 
replacement pixel values may be interpolated by processing algorithms, which convert the sensor 
data into digital image pixel values. Techniques may be employed to automatically detect and 
compensate for defective sensor pixels within a video camera (see also US patent 5,625,413 to 
Katoh et al, hereby incorporated by reference). The camera may perform a dark current 
measurement on start-up when the camera iris is closed and by raising the gain can determine 
pixels which exhibit abnormally high dark current values. The location of these pixels is 
recorded in camera memory as a LUT with associated threshold brightness values associated 
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with each pixel depending on its dark current value; defect compensation depends on input 
image brightness and ambient temperature. 

An image pickup apparatus, such as a digital camera, may have a detachable lens (see 
also US patent appHcation 2003/0133027 to Itoh, hereby incorporated by reference). The camera 
may incorporate a defect detecting section and a compensation section for sensor defects. Further 
the defect detection operation may become active when the camera lens is detached so that the 
user will not miss an opportunity to take a picture due to the operation of the defect detection 
process. 

The techniques of the preferred and alternative embodiments described herein may be 
applied to printers and to to imaging devices such as a digital cameras which incorporate a 
focusing lens system. A process may be employed for detecting and mapping dust on the surface 
of a photographic element (see also US patent 5,436,979 to Gray et al, hereby incorporated by 
reference). This may be applied in the context of a verification procedure to follow a cleaning 
process for a range of photographic elements including film negatives and sUdes. Statistical 
information may be obtained and presented to an operator to allow control of the cleaning 
process. Detailed location information may be also recorded and/or correction means may be 
also provided for dust defects on a photographic element.. 

Techniques and means to create a defect map for a digital camera or similar imaging 
device may use an all-white reference background (see also US patent application 2002/0093577 
to Kitawaki et al., hereby incorporated by reference). The location of any dust or scratch defects 
may be recorded in the memory of the imaging apparatus when the camera is in a dust detection 
mode and when a dust correction circuit is active any image data co-located with a defect may be 
corrected for the presence of dust by elimination, color correction or interpolation based on the 
surrounding pixels. Further, where the position of a dust defect changes with f-stop of the camera 
a list of dust locations corresponding to f-stop settings is pre recorded at the time of 
manufacturing in a LUT in the camera memory. Any effect of different focal length may be 
simplified to the effect of the change in dust due to magnification of the lens. In addition, 
techniques for dynamically obtaining a defect map based on the processing of a plurality of 
images may be employed with this technique. 
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Techniques may be also employed involving correcting for dust defects based on the 
geometry of said dust or of the camera. Further techniques may involve utilizing camera 
metadata to enhance the detection and correction processes for defects. Further techniques may 
involve alerting the user that the camera requires servicing due to excessive levels of dust 
contamination, or the fact that it is not only magnification but the actual lens that is mounted. 

A method of filtering dust artifacts form an acquired digital image including multiplicity 
of pixels indicative of dust, the pixels forming various shapes in the image, may be employed. 
The method may include analyzing image information including information describing 
conditions under which the image was acquired and/or acquisition device-specific information. 
One or more regions may be determined within the digital image suspected as including dust 
artifact. Based at least in part on said meta-data analysis, it may be determined whether the 
regions are actual dust artifact. 

A method may includes obtaining a dust map based on analysis of multiple images 
acquired by the same image acquisition device. The dust map may include regions of various 
shapes and sizes indicative of statistically recurring pattems associated with dust . 

A method may fiirther include analyzing the images in comparison to a predetermined 
dust map to establish the validity of the dust over progressions of time. The method may fiirther 
involve mapping the acquired image to a predetermined default acquisition condition as a 
fimction of the lens type and the focal length that was used at acquisition. 

A method may fiirther include mapping a dust spec as depicted in the dust map and the 
suspected dust specs in the acquired image based on a calculated transformation of the dust as a 
fiinction of the lens and the aperture, or other extracted parameter, used to acquire the image. 
The actual removal of dust artifacts firom an image may include a step where missing data as 
obscured by the dust specs is regenerated and in-painted based on analysis of the region in the 
image surrounding the dust spec. The actual removal of the dust artifacts firom the image may 
also include a step where deteriorated regions primarily in the periphery of the dust spec are 
enhanced and restored based on knowledge of the deterioration fimction. The actual image 
retouching may include both in-painting and restoration or either one of these operations, or 
another image correction technique as may be understood by those skilled in the art. 
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A method of detecting and removing dust artifacts may be performed in the acquisition 
device as a post-processing stage prior to saving the image. This method may further include an 
analysis of the image in its raw format immediately followed by the acquisition stage. The 
method of detecting and removing dust artifacts can be performed on an extemal device as part 
of a download or capture process. Such extemal device may be a personal computer, a storage 
device, and archival device, a display or a printing device or other device. The method of 
detecting and removing dust artifacts can be performed in part in the acquisition device and the 
extemal device. 

A dust detection and/or correction tecliiiique may be applied post priori to a collection of 
images, or individually to images as they are added to a collection. The map may be generated a 
priori to the introduction of an image, or dynamically and in concurrence to the introduction of 
new images. The method may further include steps of providing a statistical confidence level as 
to the fact that a certain region is indeed part of a dust spec. The method may further provide 
tools to determine whether the acquisition device may benefit from some maintenance. 
A method may be employed that may be implemented as part of a digitization process, such as 
correcting defects on scanning device, whether flat bed or drum, whether for hard copy 
documents or for film digitization. A method may be further applied to other recurring image 
imperfections such as dead pixels on the sensor, burnt pixels on the sensor, scratches, etc. A 
method of automatically determining whether to recommend servicing a digital image 
acquisition system including a digital camera based on dust analysis may be advantageously 
employed. A method of calculating parameters of an optical system may be based on analysis of 
the dust. While an exemplary drawings and specific embodiments of the present invention 
have been described and illustrated, it is to be understood that that the scope of the present 
invention is not to be limited to the particular embodiments discussed. Thus, the embodiments 
shall be regarded as illustrative rather than restrictive, and it should be understood that variations 
may be made in those embodiments by workers skilled in the arts without departing from the 
scope of the present invention as set forth in the claims that follow and their structural and 
functional equivalents. 



GrayCary\SFVJ093797.1 
2100874-991350 



-40- 



Express Mail: EV 330728515 US 



Atty. Docket: 2100874-991350 



In addition, in methods that may be performed according to preferred embodiments 
herein, the operations have been described in selected typographical sequences. However, the 
sequences have been selected and so ordered for typographical convenience and are not intended 
to imply any particular order for performing the operations, unless a particular ordering is 
expressly provided or understood by those skilled in the art as being necessary. 

Many references have been cited above herein, and in addition to that which is described 
as background, the invention sunmiary, brief description of the drawings, the drawings and the 
abstract, these references are hereby incorporated by reference into the detailed description of the 
preferred embodiments, as disclosing altemative embodiments of elements or features of the 
preferred embodiments not otherwise set forth in detail above. A single one or a combination of 
two or more of these references may be consulted to obtain a variation of the preferred 
embodiments described in the detailed description above. 
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